Method for encoding video, method for decoding video, and apparatus using same

ABSTRACT

The present invention relates to a method for coding a scalable video in a multilayer structure, and a method for encoding a video according to the present invention comprises the steps of: decoding and saving a picture of a reference layer; inducing an interlayer reference picture which is referenced for predicting a current block of a current layer; producing a reference picture list including the interlayer reference picture and a reference picture of the current layer; conducting a prediction on the current block of the current layer with the reference picture list to induce a prediction sample with respect to the current block; inducing a recovery sample with respect to the current block based on the prediction sample and the prediction block with respect to the current block; and transmitting reference information for indicating a picture that can be used for interlayer prediction from the pictures of the reference layer.

TECHNICAL FIELD

The present invention relates to video encoding and video decoding, andmore particularly, to a method and an apparatus for reconstructing apicture to be encoded/decoded in a current layer based on information onanother layer in a multilayer structure.

BACKGROUND ART

Recently, demands for high-resolution and high-quality pictures haveincreased in various fields of applications. As pictures have higherresolution and higher quality, the amount of information on the picturesalso increases.

With a growing amount of information, multi-functional devices andnetworks with various environments are introduced. Accordingly, the samecontent may be utilized with different levels of quality.

Specifically, as terminals are able to support diverse qualities ofpictures and various network environments are established, a picturewith general quality is enabled in one environment while ahigher-quality picture may be available in another environment.

For example, a user may enjoy video content purchased through a portableterminal on a large-screen display with higher resolution at home.

In recent years, as high definition (HD) broadcast services areavailable, a large number of users are getting used to high-resolutionand high-quality videos and service providers and service users also payattention to ultrahigh-definition (UHD) services having a resolutionfour times higher than HDTV.

Thus, there is a need to provide scalability to video quality, forexample, the image quality, resolution, size and frame rate of a video,based on high-efficiency encoding and decoding methods on ahigh-capacity video so as to offer varied qualities of video services indifferent environments for users' demands. Further, discussions onvarious video processing methods involved in scalability are alsonecessary.

SUMMARY OF THE INVENTION Technical Problems

An object of the invention is to provide a method and an apparatus forperforming effective inter-layer prediction in scalable video coding ofa multilayer structure.

Another object of the invention is to provide a method and an apparatusfor performing selective processing on pictures in a reference layer forinter-layer prediction in scalable video coding of a multilayerstructure.

Still another object of the invention is to provide a method and anapparatus for specifying reference pictures for a current layer amongpictures in a reference layer for inter-layer prediction in scalablevideo coding of a multilayer structure.

Yet another object of the invention is to provide a method and anapparatus for effectively performing inter-layer prediction based onpictures in a reference layer specified for inter-layer prediction inscalable video coding of a multilayer structure.

Technical Solution

According to an aspect of the invention, there is provided a method ofencoding scalable video in multi-layer structure, comprising deriving aninter-layer reference picture which is referred to in prediction of acurrent block in a current layer from the decoded picture in thereference layer, constructing a reference picture list comprising theinter-layer reference picture and a reference picture in the currentlayer, deriving a predicted sample of the current block by predictingthe current block in the current layer based on the reference picturelist, deriving a reconstructed sample of the current block based on thepredicted sample and a predicted block of the current block, andtransmitting reference information indicating a picture available forinter-layer prediction among pictures in the reference layer.

According to an another aspect of the invention, there is provided amethod of decoding scalable video in multi-layer structure, comprisingreceiving reference information indicating whether a picture in areference layer is available for inter-layer prediction, decoding andstoring the picture in the reference layer based on the referenceinformation, deriving an inter-layer reference picture referred to inprediction of a current block in a current layer from the decodedpicture in the reference layer based on the reference information,constructing a reference picture list comprising the inter-layerreference picture and a reference picture in the current layer, derivinga predicted sample of the current block by predicting the current blockin the current layer based on the reference picture list, and deriving areconstructed sample of the current block based on the predicted sampleand a residual sample of the current block.

Advantageous Effects

According to the present invention, effective inter-layer prediction maybe performed in scalable video coding of a multilayer structure.

According to the present invention, selective processing may beperformed on pictures in a reference layer for inter-layer prediction inscalable video coding of a multilayer structure.

According to the present invention, reference pictures for a currentlayer may be specified among pictures in a reference layer forinter-layer prediction in scalable video coding of a multilayerstructure.

According to the present invention, inter-layer prediction mayeffectively be performed based on pictures in a reference layerspecified for inter-layer prediction in scalable video coding of amultilayer structure.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram schematically illustrating a video encoderthat supports scalability according to an embodiment of the invention.

FIG. 2 is a block diagram illustrating an example of inter-layerprediction in an encoder which performs scalable coding according to thepresent invention.

FIG. 3 schematically illustrates an example of a sequence structure oftwo spatial layers according to the present embodiment.

FIG. 4 is a diagram schematically illustrating the present embodiment,which shows an example of a method of removing coded pictures that donot need transmitting to a decoder or decoding.

FIG. 5 schematically illustrates an example of memory management.

FIG. 6 schematically illustrates an example of memory managementaccording to the present invention.

FIG. 7 schematically illustrates a case in which pictures not needed fordecoding are excluded in the example of FIG. 6.

FIG. 8 schematically illustrates an example of content/memory state/DPBstate in decoding each picture.

FIG. 9 is a flowchart schematically illustrating an example of anoperation of a video encoder according to the present invention.

FIG. 10 is a flowchart schematically illustrating an example of anoperation of a video decoder according to the present invention.

DESCRIPTION OF EMBODIMENTS

The present invention can be modified in various forms, and specificembodiments thereof will be described and shown in the drawings.However, the embodiments are not intended for limiting the invention.The terms used in the following description are used to merely describespecific embodiments, but are not intended to limit the invention. Anexpression of a singular number includes an expression of the pluralnumber, so long as it is clearly read differently. The terms such as“include” and “have” are intended to indicate that features, numbers,steps, operations, elements, components, or combinations thereof used inthe following description exist and it should be thus understood thatthe possibility of existence or addition of one or more differentfeatures, numbers, steps, operations, elements, components, orcombinations thereof is not excluded.

On the other hand, elements in the drawings described in the inventionare independently drawn for the purpose of convenience for explanationof different specific functions in an image encoding/decoding apparatusand does not mean that the elements are embodied by independent hardwareor independent software. For example, two or more elements of theelements may be combined to form a single element, or one element may bedivided into plural elements. The embodiments in which the elements arecombined and/or divided belong to the scope of the invention withoutdeparting from the concept of the invention.

Hereinafter, exemplary embodiments of the invention will be described indetail with reference to the accompanying drawings. Like constituents inthe drawings will be referenced by like reference numerals and will notbe repeatedly described.

In a video coding method that supports scalability (hereinafter,referred to as “scalable coding”), input signals can be processed bylayers. Depending on the layers, the input signals (input videos) may bedifferent from each other in at least one of a resolution, a frame rate,a bit depth, a color format, and an aspect ratio.

In this description, scalable coding includes scalable encoding andscalable decoding.

In scalable encoding/decoding, it is possible to reduce duplicatetransmission/processing of information and to enhance compressionefficiency by performing inter-layer prediction using an inter-layerdifference, that is, on the basis of scalability.

FIG. 1 is a block diagram schematically illustrating a video encoderthat supports scalability according to an embodiment of the invention.

FIG. 1 illustrates a multilayer structure including two layers forconvenience of description. However, the present invention is notlimited thereto and a multilayer structure according to the presentinvention may include two or more layers.

Referring to FIG. 1, the video encoder 100 includes an encoding module105 for layer 1 and an encoding module 135 for layer 0.

Layer 0 may be a base layer, a reference layer or a lower layer, andlayer 1 may be an enhancement layer, a current layer or an upper layer.

The encoding module 105 for layer 1 includes a prediction module 110, atransform/quantization module 115, a filtering module 120, a decodedpicture buffer (DPB) 125, an entropy coding module 130, and amultiplexer (MUX) 165.

The encoding module 135 for layer 0 includes a prediction module 140, atransform/quantization module 145, a filtering module 150, a DPB 155,and an entropy coding module 160.

The prediction modules 110 and 140 may perform inter prediction andintra prediction on an input video. The prediction modules 110 and 140may perform the predictions by predetermined processing units. Theprocessing unit for prediction may be a coding unit (CU), a predictionunit (PU), or may be a transform unit (TU).

For example, the prediction modules 110 and 140 may determine whether toconduct inter prediction or intra prediction by CU, may determine aprediction mode by PU, and may perform prediction by PU or TU.Prediction to be performed includes construction of a predicted blockand construction of a residual block (residual signal).

In the inter prediction, the prediction may be performed on the basis ofinformation on at least one of a previous picture and/or a subsequentpicture of a current picture to construct a predicted block. In theintra prediction, the prediction may be performed on the basis ofinformation on a pixel in a current picture to construct a predictedblock.

Examples of an inter prediction mode or method include a skip mode, amerge mode, a motion vector prediction (MVP) method. In the interprediction, a reference picture for a current PU to be predicted may beselected and a reference block corresponding to the current PU may beselected from the reference picture. The prediction modules 110 and 140may construct a predicted block on the basis of the reference block.

The predicted block may be constructed as an integer sample unit or as afractional pixel unit. Here, a motion vector may also be represented ina fractional pixel.

Motion information in the inter prediction, that is, information such asan index, a motion vector and a residual signal of a reference picture,is entropy-encoded and is transmitted to a video decoder. When a skipmode is applied, the residual signal may not be created, transformed,quantized, and transmitted at all.

Prediction modes in the intra prediction may include 33 directionalprediction modes and at least two non-directional modes. Thenon-directional modes may include a DC prediction mode and a planarmode. In the intra prediction, a predicted block may be constructedafter a filter is applied to a reference sample.

A PU may be a block with various sizes and shapes. For example, in interprediction, a PU may be a 2N×2N, 2N×N, N×2N or N×N block (where N is aninteger). In intra prediction, a PU may be a 2N×2N or N×N block (where Nis an integer). A PU with a size of N×N may be set to be applied only toa specific case. For example, the PU with the size of N×N may be set tobe used only for a smallest CU or only for intra prediction. In additionto the PUs with the above-mentioned sizes, a PU may be further definedas an N×mN block, an mN×N block, a 2N×mN block, or an mN×2N block (wherem<1) for use.

The prediction modules 110 may perform prediction for layer 1 usinginformation on layer 0. In this specification, a process of predictingcurrent layer information using another layer information is defined asan inter-layer prediction for convenience.

The current layer information predicted using the other layerinformation (that is, predicted by the inter-layer prediction) mayinclude at least one selected from a texture, motion information, unitinformation, a predetermined parameter (for example, a filteringparameter).

The other layer information used for predicting the current layerinformation (that is, used for the inter-layer prediction) may includeat least one selected form a texture, motion information, unitinformation, a predetermined parameter (for example, a filteringparameter).

In inter-layer prediction, a current block is a block in a currentpicture in a current layer (layer 1 in FIG. 1), which may be a block tobe encoded. A reference block is a block in a picture (referencepicture) belonging to the same access unit (AU) as the picture (currentpicture) including the current block, which may be a block correspondingto the current block in a layer (reference layer, layer 0 in FIG. 1)which is referred to in prediction of the current block.

An example of inter-layer prediction includes inter-layer motionprediction which predicts motion information on a current layer usingmotion information on a reference layer. According to inter-layer motionprediction, motion information on a current block may be predicted usingmotion information on a reference block.

When inter-layer motion prediction is applied, the prediction module 110may scale and use motion information on a reference layer.

As another example of inter-layer prediction, inter-layer textureprediction may use a texture of a reconstructed reference block as thepredictive value of a current block. Here, the prediction module 110 mayscale the texture of the reference block by upsampling.

As still another example of inter-layer prediction, inter-layer unitinformation prediction may derive unit (CU, PU and/or TU) information ona reference layer to use as unit information on a current layer, ordetermine unit information on the current layer based on unitinformation on the reference layer.

Unit information may include information in each unit level. Forinstance, CU information may include information on partition (CU, PUand/or TU), information on transformation, information on prediction,and information on coding. PU information may include information on PUpartition and information on prediction (for example, motion informationand information on a prediction mode). TU information may includeinformation on TU partition and information on transformation (transformcoefficients and transform methods).

As yet another example of inter-layer prediction, inter-layer parameterprediction may reuse a derived parameter of a reference layer for acurrent layer or predict a parameter for the current layer based on theparameter used for the reference layer.

As still another example of inter-layer prediction, inter-layer residualprediction may predict a residual of a current layer using residualinformation on another layer and predict a current block based on theresidual of the current layer.

As yet another example of inter-layer prediction, inter-layerdifferential prediction may predict a current block using a differentialbetween pictures obtained by upsampling or downsampling a reconstructedpicture in a current layer and a reconstructed picture in a referencelayer.

As still another example of inter-layer prediction, inter-layer syntaxprediction may predict or generate a texture of a current block usingsyntax information on a reference layer. Here, the syntax information onthe reference layer used for reference may include information on anintra prediction mode, motion information and the like.

A plurality of inter-layer prediction methods among the afore-describedmethods may be used in prediction of a particular block.

Although inter-layer texture prediction, inter-layer motion prediction,inter-layer unit information prediction, inter-layer parameterprediction, inter-layer residual prediction, inter-layer differentialprediction, and inter-layer syntax prediction have been described asexamples of inter-layer prediction, inter-layer prediction applicable tothe present invention is not limited thereto.

For instance, inter-layer prediction may be applied as an extension ofinter prediction for a current layer. That is, a reference picturederived from a reference layer may be included in reference picturesavailable for reference in inter prediction of a current block toconduct inter prediction of the current block.

In this case, an inter-layer reference picture may be included in areference picture list for the current block. The prediction module 110may perform inter prediction of the current block using the inter-layerreference picture.

Here, the inter-layer reference picture may be a reference pictureconstructed by sampling a reconstructed picture in the reference layerto correspond to the current layer. Thus, when the reconstructed picturein the reference layer corresponds to a picture in the current layer,the reconstructed picture in the reference layer may be used as aninter-layer reference picture without sampling. For instance, whensamples of the reconstructed picture in the reference layer and thereconstructed picture in the current layer have the same width andheight and offsets between the picture in the reference layer and thepicture in the current layer at the top left, top right, bottom left andbottom right are 0, the reconstructed picture in the reference layer maybe used as an inter-layer reference picture for the current layerwithout being subjected to sampling.

The reconstructed picture in the reference layer from which theinter-layer reference picture is derived may belong to the same AU asthe current picture to be encoded.

When inter prediction of the current block is performed with thereference picture list including the inter-layer reference picture, theposition of the inter-layer reference picture in the reference picturelist may vary in reference picture lists L0 and L1. For instance, inreference picture list L0, the inter-layer reference picture may bepositioned subsequent to short-term reference pictures prior to thecurrent picture. In reference picture list L1, the inter-layer referencepicture may be positioned last.

Here, reference picture list L0 is a reference picture list used forinter prediction of a predictive slice (“P slice”) or used as a firstreference picture list in inter prediction of a bi-predictive slice (“Bslice”). Reference picture list L1 is a second reference picture listused in inter prediction of the B slice.

Thus, reference picture list L0 may be constructed by sequentiallyincluding short-term reference picture(s) prior to the current picture,the inter-layer reference picture, short-term reference picture(s)subsequent to the current picture, and a long-term reference picture.Reference picture list L1 may be constructed by sequentially includingshort-term reference picture(s) subsequent to the current picture,short-term reference picture(s) prior to the current picture, along-term reference picture, and the inter-layer reference picture.

Here, a P slice is a slice to be subjected to intra prediction or tointer prediction using at most one motion vector per prediction blockand a reference picture index. A B slice is a slice to be subjected tointra prediction or to prediction using at most two motion vectors perprediction block and a reference picture index. Further, an intra slice(“I slice”) is a slice subjected only to intra prediction.

A slice is a sequence of one or more slice segments. A slice sequencestarts from an independent slice segment. When dependent slice segmentspreceding a next independent slice segment are present in the same AU, aslice includes an independent slice segment as the starter of a slicesequence and the dependent slice segments prior to the next independentslice.

Slice segments may be a sequence of coding tree units (CTUs) or codingtree blocks (CTBs) consecutively ordered in a tile scan and included ina single network abstraction layer (NAL) unit. A CTU is a coding unit ina quadtree structure, which may be a largest coding unit (LCU). In thisspecification, the terms “CTU” and “LCU” may collectively be used asnecessary for better understanding of the invention.

In a slice segment, a first CTB (CTU) of a slice segment or a portionincluding a data element about all CTBs (CTUs) is referred to as a slicesegment head. Here, a slice segment head of an independent slice segmentis referred to as a slice header.

A slice may be a transfer unit of an NAL unit. For example, an NAL unitmay constructed by including slices or slice segments.

When inter prediction of the current block is performed based on thereference picture list including the inter-layer reference picture, thereference picture list may include a plurality of inter-layer referencepictures derived from a plurality of layers.

When the reference picture list includes a plurality of inter-layerreference pictures, the inter-layer reference pictures may be arrangedin an interchanged manner in L0 and L1. For example, suppose that twointer-layer reference pictures, inter-layer reference picture ILRPi andinter-layer reference picture ILRPj, are included in the referencepicture list used for inter prediction of the current block. In thiscase, ILRPi may be positioned subsequent to short-term referencepictures prior to the current picture and ILRPj may be positioned lastin reference picture list L0. Also, ILRPi may be positioned last andILRPj may be positioned subsequent to short-term reference picturesprior to the current picture in reference picture list L1.

In this case, reference picture list L0 may be constructed bysequentially including short-term reference picture(s) prior to thecurrent picture, inter-layer reference picture ILRPi, short-termreference picture(s) subsequent to the current picture, a long-termreference picture, and inter-layer reference picture ILRPj. Referencepicture list L1 may be constructed by sequentially including short-termreference picture(s) subsequent to the current picture, inter-layerreference picture ILRPj, short-term reference picture(s) prior to thecurrent picture, a long-term reference picture, and inter-layerreference picture ILRPi.

Further, one of the two inter-layer reference pictures may be aninter-layer reference picture derived from a resolution scalable layer,and the other may be an inter-layer reference picture derived from alayer providing a different view. In this case, for instance, when ILRPiis an inter-layer reference picture derived from a layer providing adifferent level of resolution and ILRPj is an inter-layer referencepicture derived from a layer providing a different view, in scalablevideo coding supporting only other scalabilities than view scalability,reference picture list L0 may be constructed by sequentially includingshort-term reference picture(s) prior to the current picture,inter-layer reference picture ILRPi, short-term reference picture(s)subsequent to the current picture and a long-term reference picture, andreference picture list L1 may be constructed by sequentially includingshort-term reference picture(s) subsequent to the current picture,short-term reference picture(s) prior to the current picture, along-term reference picture, and inter-layer reference picture ILRPi.

Meanwhile, in inter-layer prediction, as information on the inter-layerreference picture, only a sample value may be used, only motioninformation (motion vector) may be used, or both a sample value andmotion information may be used. When a reference picture index indicatesthe inter-layer reference picture, the prediction module 110 may useonly the sample value of the inter-layer reference picture, only themotion information (motion vector) on the inter-layer reference picture,or both the sample value of the inter-layer reference picture and themotion information on the inter-layer reference picture depending oninformation received from the encoder.

When only the sample value of the inter-layer reference picture is used,the prediction module 110 may derive samples of a block specified by themotion vector in the inter-layer reference picture as a predicted sampleof the current block. In scalable video coding which does not consider aview, a motion vector in inter prediction using an inter-layer referencepicture (inter-layer prediction) may be set to a fixed value (forexample, 0).

When only the motion information on the inter-layer reference picture isused, the prediction module 110 may use a motion vector specified in theinter-layer reference picture as a motion vector predictor for derivinga motion vector of the current block. Further, the prediction module 110may use the motion vector specified in the inter-layer reference pictureas the motion vector of the current block.

When both the sample of the inter-layer reference picture and the motioninformation on the inter-layer reference picture are used, theprediction module 110 may use a sample in a region corresponding to thecurrent block in the inter-layer reference picture and motioninformation (motion vector) specified in the inter-layer referencepicture for prediction of the current block.

When inter-layer prediction is applied, the encoder may transmit thereference index indicating the inter-layer reference picture in thereference picture list to a decoder, and also transmit informationspecifying which information (sample information, motion information orboth sample information and motion information) on the inter-layerreference picture the encoder uses, that is, information specifying thetype of dependency between two layers in inter-layer prediction, to thedecoder.

The transform/quantization modules 115 and 145 may transform theresidual block by TU to create transform coefficients and may quantizethe transform coefficients.

A transform block is a rectangular block of samples to which the sametransformation is applied. The transform block may be a TU and may havea quad-tree structure.

The transform/quantization modules 115 and 145 may performtransformation based on a prediction mode applied to the residual blockand a size of the transform block and a size of the transform block tocreate a two-dimensional (2D) array of transform coefficients. Forexample, when intra prediction is applied to the residual block and theresidual block has a 4×4 array, the residual block may be transformedusing discrete sine transform (DST). Otherwise, the residual block maybe transformed using discrete cosine transform (DCT).

The transform/quantization modules 115 and 145 may quantize thetransform coefficients to create the quantized transform coefficients.

The transform/quantization modules 115 and 145 may transmit thequantized transform coefficients to the entropy coding modules 130 and160. Here, the transform/quantization modules 115 and 145 may rearrangethe 2D array of the quantized transform coefficients into aone-dimensional (1D) array in a predetermined scan order and maytransmit the rearranged 1D array to the entropy coding modules 130 and160. The transform/quantization modules 115 and 145 may transmit areconstructed block generated on the basis of the residual block and thepredicted block to the filtering modules 120 and 150 for interprediction, without being transformation/quantization.

If necessary, the transform/quantization modules 115 and 145 may skiptransformation and perform only quantization or may skip bothtransformation and quantization. For example, the transform/quantizationmodules 115 and 165 may skip transformation for a block involving aspecific prediction method or having a specific size, or a blockinvolving a specific prediction block and having a specific size.

The entropy coding modules 130 and 160 may perform entropy encoding onthe quantized transform coefficients. An encoding method, such asexponential Golomb coding and context-adaptive binary arithmetic coding(CABAC), may be used for entropy encoding.

The filtering modules 120 and 150 may apply a deblocking filter, anadaptive loop filter (ALF), or a sample adaptive offset (SAO) to areconstructed picture.

The deblocking filter may remove a block distortion generated onboundaries between blocks in the reconstructed picture. The ALF mayperform a filtering process on the basis of a resulting value ofcomparing the original picture with the reconstructed picture of whichthe blocks are filtered by the deblocking filter. The SAO mayreconstruct an offset difference by pixel between the residual blockhaving been subjected to the deblocking filter and the original pictureand is applied in a form of a band offset, an edge offset, or the like.

The filtering modules 120 and 150 may not apply all of the deblockingfilter, the ALF and the SAO, but may apply only the deblocking filter,may apply only the deblocking filter and the ALF, or may apply only thedeblocking filter and the SAO.

The DPBs 125 and 155 may receive and store the reconstructed block orthe reconstructed picture from the filtering modules 125 and 150. TheDPB 125 and 155 may provide the reconstructed block or picture to theprediction modules 110 and 140 that perform inter prediction.

Information output from the entropy coding module 160 for layer 0 andinformation output from the entropy coding module 130 for layer 1 may bemultiplexed by the MUX 165 and may be output as a bitstream.

Although the encoding module 105 for layer 1 has been described toinclude the MUX 165 for convenience, the MUX may be a device or moduleindependent of the encoding module 105 for layer 1 and the encodingmodule 135 for layer 0.

Further, although it has been described that scalable video coding isperformed in a multilayer structure including two layers, the presentinvention is not limited thereto. For example, the encoder of FIG. 1 mayalso be applied to a multilayer structure including two or more layers.When the encoder is applied to a multilayer structure including Nlayers, layer 0 and layer 1 of FIG. 1 may be two layers having areference relationship among the N layers.

Here, the encoder may transmit information specifying the referencerelationship between the two layers to the decoder. For example, whenthe two layers have a dependent relationship in inter-layer prediction,the encoder may transmit direct_dependency_flag[L][M]. Whendirect_dependency_flag[L][M] is 1, an Lth layer may be predicted byreferring to an Mth layer.

FIG. 2 is a block diagram illustrating an example of inter-layerprediction in an encoder which performs scalable coding according to thepresent invention.

Referring to FIG. 2, the video decoder 200 includes a decoding module210 for layer 1 and a decoding module 250 for layer 0.

Layer 0 may be a base layer, a reference layer or a lower layer, andlayer 1 may be an enhancement layer, a current layer or an upper layer.

The decoding module 210 for layer 1 may include an entropy decodingmodule 215, a rearrangement module 220, a dequantization module 225, aninverse transform module 230, a prediction module 235, a filteringmodule 240, and a memory.

The decoding module 250 for layer 0 may include an entropy decodingmodule 255, a rearrangement module 260, a dequantization module 265, aninverse transform module 270, a filtering module 280, and a memory 285.

When a bitstream including video information is transmitted from thevideo encoder, a demultiplexer (DEMUX) 305 may demultiplex theinformation by layers and may transmit the information to decoders bylayers.

The entropy decoding modules 215 and 255 may perform entropy decodingcorresponding to an entropy coding method used in the video encoder. Forexample, when CABAC is used in the video encoder, the entropy decodingmodules 215 and 255 may perform entropy decoding using CABAC.

Information for constructing a predicted block out of informationdecoded by the entropy decoding modules 215 and 255 may be provided tothe prediction modules 235 and 275, and residual values entropy-decodedby the entropy decoding modules 215 and 255, that is, quantizedtransform coefficients, may be input to the rearrangement modules 220and 260.

The rearrangement modules 220 and 260 may rearrange the information ofthe bitstream entropy-decoded by the entropy decoding modules 215 and255, that is, the quantized transform coefficients, on the basis of arearrangement method used in the video encoder.

For example, the rearrangement modules 220 and 260 may rearrange a 1Darray of the quantized transform coefficients back into a 2D array ofcoefficients. The rearrangement modules 220 and 260 may perform scanningon the basis of a prediction mode applied to a current block (transformblock) and/or a size of the transform block to construct a 2D array ofcoefficients (quantized transform coefficients).

The dequantization modules 225 and 265 may perform dequantization on thebasis of a quantization parameter transmitted from the video encoder andthe rearranged coefficients of the block to create transformcoefficients.

The dequantization modules 225 and 265 may transmit the entropy-decodedresidual values to the inverse transform modules 230 and 270, withoutdequantizing the residual values, depending on a predetermined conditionor depending on a quantization method used for the video encoder.

In the video encoder, DCT and/or DST may be selectively performeddepending on a plurality of information pieces, such as a predictionmethod, a size of a current block and a prediction direction, and theinverse transform modules 230 and 270 of the video decoder may performinverse transformation on the basis of transform information used by thevideo decoder.

For example, the inverse transform modules 230 and 270 may performinverse DCT and inverse DST depending on a prediction mode/block size.Specifically, the inverse transform modules 230 and 270 may performinverse DST on a 4×4 luma block to which intra prediction has beenapplied.

Alternatively, the inverse transform modules 230 and 270 may fixedly usea specific inverse transformation method regardless of a predictionmode/block size. For example, the inverse transform modules 230 and 270may apply only inverse DST to all transform blocks. The inversetransform modules 230 and 270 may also apply only inverse DCT to alltransform blocks.

The inverse transform modules 230 and 270 may inversely transform thetransform coefficients or a block of the transform coefficients toconstruct a residual signal or a residual block.

The inverse transform modules 230 and 270 may skip transformation ifnecessary or depending on an encoding method used for the video encoder.For example, the inverse transform modules 230 and 270 may skiptransformation for a block involving a specific prediction method orhaving a specific size, or a block involving a specific prediction blockand having a specific size.

The prediction modules 235 and 275 may construct a predicted block ofthe current block on the basis of predicted block constructioninformation provided from the entropy decoding modules 215 and 255 andinformation on a previously decoded block and/or picture provided fromthe memories 245 and 285.

When a prediction mode for the current block is an intra predictionmode, the prediction modules 235 and 275 may perform intra prediction onthe current block on the basis of information on a pixel in a currentpicture.

When the prediction mode for the current block is an inter predictionmode, the prediction modules 235 and 275 may perform inter prediction onthe current block on the basis of information included in at least oneof a previous picture and a subsequent picture of the current picture.Part or all of motion information necessary for inter prediction may bederived based on information received from the video encoder.

When the skip mode is used as an inter prediction mode, the residual maynot be transmitted from the video encoder and the predicted block may beused as a reconstructed block.

The prediction module 235 for layer 1 may perform inter prediction orintra prediction using only information in layer 1 and may performinter-layer prediction using information on another layer (layer 0).

Information on a current layer predicted using information on anotherlayer (that is, predicted by inter-layer prediction) may be at least oneof a texture, motion information, unit information, and a predeterminedparameter (for example, a filtering parameter).

The information on another layer information used for prediction of thecurrent layer (that is, used for inter-layer prediction) may be at leastone of a texture, motion information, unit information, and apredetermined parameter (for example, a filtering parameter).

In inter-layer prediction, a current block is a block in a currentpicture in a current layer (layer 1 in FIG. 2), which may be a block tobe decoded. A reference block is a block in a picture (referencepicture) belonging to the same access unit (AU) as the picture (currentpicture) including the current block, which may be a block correspondingto the current block in a layer (reference layer, layer 0 in FIG. 2)which is referred to in prediction of the current block.

An example of inter-layer prediction includes inter-layer motionprediction which predicts motion information on a current layer usingmotion information on a reference layer. According to inter-layer motionprediction, motion information on a current block may be predicted usingmotion information on a reference block.

When inter-layer motion prediction is applied, the prediction module 235may scale and use motion information on a reference layer.

As another example of inter-layer prediction, inter-layer textureprediction may use a texture of a reconstructed reference block as thepredictive value of a current block. Here, the prediction module 235 mayscale the texture of the reference block by upsampling.

As still another example of inter-layer prediction, inter-layer unitinformation prediction may derive unit (CU, PU and/or TU) information ona reference layer to use as unit information on a current layer, ordetermine unit information on the current layer based on unitinformation on the reference layer.

Unit information may include information in each unit level. Forinstance, CU information may include information on partition (CU, PUand/or TU), information on transformation, information on prediction,and information on coding. PU information may include information on PUpartition and information on prediction (for example, motion informationand information on a prediction mode). TU information may includeinformation on TU partition and information on transformation (transformcoefficients and transform methods).

As yet another example of inter-layer prediction, inter-layer parameterprediction may reuse a derived parameter of a reference layer for acurrent layer or predict a parameter for the current layer based on theparameter used for the reference layer.

As still another example of inter-layer prediction, inter-layer residualprediction may predict a residual of a current layer using residualinformation on another layer and predict a current block based on theresidual of the current layer.

As yet another example of inter-layer prediction, inter-layerdifferential prediction may predict a current block using a differentialbetween pictures obtained by upsampling or downsampling a reconstructedpicture in a current layer and a reconstructed picture in a referencelayer.

As still another example of inter-layer prediction, inter-layer syntaxprediction may predict or generate a texture of a current block usingsyntax information on a reference layer. Here, the syntax information onthe reference layer used for reference may include information on anintra prediction mode and motion information.

A plurality of inter-layer prediction methods among the afore-describedmethods may be used in prediction of a particular block.

Although inter-layer texture prediction, inter-layer motion prediction,inter-layer unit information prediction, inter-layer parameterprediction, inter-layer residual prediction, inter-layer differentialprediction, and inter-layer syntax prediction have been described asexamples of inter-layer prediction, inter-layer prediction applicable tothe present invention is not limited thereto.

For instance, inter-layer prediction may be applied as an extension ofinter prediction for a current layer. That is, a reference picturederived from a reference layer may be included in reference picturesavailable for reference in inter prediction of a current block toconduct inter prediction of the current block.

The prediction module 235 may perform inter-layer prediction using aninter-layer reference picture when a reference picture index receivedfrom an encoder or derived from a neighboring block indicates theinter-layer reference picture in a reference picture list. For example,when the reference picture index indicates the inter-layer referencepicture, the prediction module 235 may derive the sample value of aregion specified by a motion vector in a reference picture as apredicted block of the current block.

In this case, an inter-layer reference picture may be included in areference picture list for the current block. The prediction module 235may perform inter prediction of the current block using the inter-layerreference picture.

Here, the inter-layer reference picture may be a reference pictureconstructed by sampling a reconstructed picture in the reference layerto correspond to the current layer. Thus, when the reconstructed picturein the reference layer corresponds to a picture in the current layer,the reconstructed picture in the reference layer may be used as aninter-layer reference picture without sampling. For instance, whensamples of the reconstructed picture in the reference layer and thereconstructed picture in the current layer have the same width andheight and offsets between the picture in the reference layer and thepicture in the current layer at the top left, top right, bottom left andbottom right are 0, the reconstructed picture in the reference layer maybe used as an inter-layer reference picture for the current layerwithout being subjected to sampling.

The reconstructed picture in the reference layer from which theinter-layer reference picture is derived may belong to the same AU asthe current picture to be encoded. When inter prediction of the currentblock is performed with the reference picture list including theinter-layer reference picture, the position of the inter-layer referencepicture in the reference picture list may vary in reference picturelists L0 and L1. For instance, in reference picture list L0, theinter-layer reference picture may be positioned subsequent to short-termreference pictures prior to the current picture. In reference picturelist L1, the inter-layer reference picture may be positioned last.

Here, reference picture list L0 is a reference picture list used forinter prediction of a P slice or used as a first reference picture listin inter prediction of a B slice. Reference picture list L1 is a secondreference picture list used in inter prediction of the B slice.

Thus, reference picture list L0 may be constructed by sequentiallyincluding short-term reference picture(s) prior to the current picture,the inter-layer reference picture, short-term reference picture(s)subsequent to the current picture, and a long-term reference picture.Reference picture list L1 may be constructed by sequentially includingshort-term reference picture(s) subsequent to the current picture,short-term reference picture(s) prior to the current picture, along-term reference picture, and the inter-layer reference picture.

Here, a P slice is a slice to be subjected to intra prediction or tointer prediction using at most one motion vector per prediction blockand a reference picture index. A B slice is a slice to be subjected tointra prediction or to prediction using at most two motion vectors perprediction block and a reference picture index. Further, an I slice is aslice subjected only to intra prediction.

When inter prediction of the current block is performed based on thereference picture list including the inter-layer reference picture, thereference picture list may include a plurality of inter-layer referencepictures derived from a plurality of layers.

When the reference picture list includes a plurality of inter-layerreference pictures, the inter-layer reference pictures may be arrangedin an interchanged manner in L0 and L1. For example, suppose that twointer-layer reference pictures, inter-layer reference picture ILRPi andinter-layer reference picture ILRPj, are included in the referencepicture list used for inter prediction of the current block. In thiscase, ILRPi may be positioned subsequent to short-term referencepictures prior to the current picture and ILRPj may be positioned lastin reference picture list L0. Also, ILRPi may be positioned last andILRPj may be positioned subsequent to short-term reference picturesprior to the current picture in reference picture list L1.

In this case, reference picture list L0 may be constructed bysequentially including short-term reference picture(s) prior to thecurrent picture, inter-layer reference picture ILRPi, short-termreference picture(s) subsequent to the current picture, a long-termreference picture, and inter-layer reference picture ILRPj. Referencepicture list L1 may be constructed by sequentially including short-termreference picture(s) subsequent to the current picture, inter-layerreference picture ILRPj, short-term reference picture(s) prior to thecurrent picture, a long-term reference picture, and inter-layerreference picture ILRPi.

Further, one of the two inter-layer reference pictures may be aninter-layer reference picture derived from a resolution scalable layer,and the other may be an inter-layer reference picture derived from alayer providing a different view. In this case, for instance, when ILRPiis an inter-layer reference picture derived from a layer providing adifferent level of resolution and ILRPj is an inter-layer referencepicture derived from a layer providing a different view, in scalablevideo coding supporting only other scalabilities than view scalability,reference picture list L0 may be constructed by sequentially includingshort-term reference picture(s) prior to the current picture,inter-layer reference picture ILRPi, short-term reference picture(s)subsequent to the current picture and a long-term reference picture, andreference picture list L1 may be constructed by sequentially includingshort-term reference picture(s) subsequent to the current picture,short-term reference picture(s) prior to the current picture, along-term reference picture, and inter-layer reference picture ILRPi.

Meanwhile, in inter-layer prediction, as information on the inter-layerreference picture, only a sample value may be used, only motioninformation (motion vector) may be used, or both a sample value andmotion information may be used. When a reference picture index indicatesthe inter-layer reference picture, the prediction module 235 may useonly the sample value of the inter-layer reference picture, only themotion information (motion vector) on the inter-layer reference picture,or both the sample value of the inter-layer reference picture and themotion information on the inter-layer reference picture depending oninformation received from the encoder.

When only the sample value of the inter-layer reference picture is used,the prediction module 235 may derive samples of a block specified by themotion vector in the inter-layer reference picture as a predicted sampleof the current block. In scalable video coding which does not consider aview, a motion vector in inter prediction using an inter-layer referencepicture (inter-layer prediction) may be set to a fixed value (forexample, 0).

When only the motion information on the inter-layer reference picture isused, the prediction module 235 may use a motion vector specified in theinter-layer reference picture as a motion vector predictor for derivinga motion vector of the current block. Further, the prediction module 235may use the motion vector specified in the inter-layer reference pictureas the motion vector of the current block.

When both the sample of the inter-layer reference picture and the motioninformation on the inter-layer reference picture are used, theprediction module 235 may use a sample in a region corresponding to thecurrent block in the inter-layer reference picture and motioninformation (motion vector) specified in the inter-layer referencepicture for prediction of the current block.

The decoder may receive the reference index indicating the inter-layerreference picture in the reference picture list from the encoder andperform inter-layer prediction based on the reference index. Also, thedecoder may receive information specifying which information (sampleinformation, motion information or both sample information and motioninformation) on the inter-layer reference picture the decoder uses, thatis, information specifying the type of dependency between two layers ininter-layer prediction, from the encoder.

Adders 290 and 295 may construct a reconstructed block using thepredicted block constructed by the prediction modules 235 and 275 andthe residual block constructed by the inverse transform modules 230 and270. In this case, the adders 290 and 295 may be considered as separatemodules (reconstructed block constructing module) that construct areconstructed block.

The block and/or picture reconstructed by the adders 290 and 295 may besupplied to the filtering modules 240 and 280.

Referring to FIG. 2, the filtering module 240 for layer 1 may perform afiltering operation on the reconstructed picture using the parameterinformation transmitted from the prediction module 235 for layer 1and/or the filtering module 280 for layer 1. For example, the filteringmodule 240 for layer 1 may perform a filtering operation on layer 1 oran inter-layer filtering operation using a parameter predicted from afiltering parameter applied to layer 0.

The memories 245 and 285 may store the reconstructed block or picturefor use as a reference picture or reference block. The memories 245 and285 may output the reconstructed picture stored in the memories 245 and285 via a predetermined output module (not shown) or a display (notshown).

Although FIG. 2 illustrates the rearrangement modules, thedequantization modules and the inverse transform modules as independentmodules, the video decoder may also be configured to enable thedequantization/inverse transform modules as a single module tosequentially perform rearrangement, dequantization, and inversetransform like the video encoder of FIG. 1.

Further, although it has been described that scalable video decoding isperformed in a multilayer structure including two layers, the presentinvention is not limited thereto. For example, the decoder of FIG. 2 mayalso be applied to a multilayer structure including two or more layers.When the encoder is applied to a multilayer structure including Nlayers, layer 0 and layer 1 of FIG. 2 may be two layers having areference relationship among the N layers.

Here, the decoder may receive information specifying the referencerelationship between the two layers from the encoder. For example, whenthe two layers have a dependent relationship in inter-layer prediction,the decoder may receive direct_dependency_flag[L][M]. Whendirect_dependency_flag[L][M] is 1, an Lth layer may be predicted byreferring to an Mth layer.

Although FIGS. 1 and 2 illustrate the prediction modules, the predictionmodule for layer 1 may include an inter-layer prediction module thatperforms a prediction process using information on another layer (layer0) and an inter/intra prediction module that performs a predictionprocess without using information on another layer (layer 0).

In encoding and decoding of a video supporting a plurality of layers ina bistream, that is, scalable coding, there are strong correlationsamong the plurality of layers. Thus, when prediction is performed usingthe correlations, redundant elements of data may be removed and videoencoding performance may be enhanced.

A plurality of layers may be different from one another in at least oneof resolution, frame rate, color format and view. Thus, upsampling ordownsampling for adjusting resolution may be performed to useinformation on another layer in inter-layer prediction.

Inter-layer dependency between different layers indicates whether ablock/picture in one layer is decoded by referring to a block/picture inanother layer. Thus, in a multilayer structure, when there is dependencybetween a current layer and a reference layer, the current layer may bepredicted by referring to the reference layer. A “layer” may be arepresentation of a coding dimension. A coding dimension may includescalability (for example, spatial scalability, temporal scalability andquality scalability), a view and a depth coding representation. It isregarded that inter-layer dependency is considered when inter-layerprediction is applied, without being limited thereto. For example,inter-layer dependency may also be considered in inter-layer intraprediction, inter-layer motion prediction, inter-layer syntaxprediction, inter-layer residual prediction and inter-layer unitprediction which are described above.

When a picture in a particular layer is processed, the encoder mayselect whether to use inter-layer prediction based on rate anddistortion (R-D) performance.

FIG. 3 schematically illustrates an example of a sequence structure oftwo spatial layers according to the present embodiment.

A spatial layer refers to a layer in which spatial scalability issupported.

Referring to FIG. 3, black bold arrows indicate temporal dependencybetween pictures in the same spatial layer. In FIG. 3, red dotted arrowsindicate inter-layer dependency between pictures in different spatiallayers.

In the example of FIG. 3, x^(y) represents an xth picture in layer y.

FIG. 3 shows that inter-layer prediction is applied to picture 0,picture 3, picture 6, and the like. In the structure illustrated in FIG.3, the encoding order of pictures and delivery order of the picturesfrom an encoder to a decoder are the same as order 1 as follows.

Order 1: 0⁰ 0¹ 1⁰ 1¹ 2⁰ 2¹ 3⁰ 3¹ 4⁰ 4¹ 5⁰ 5¹ 6⁰ 6¹ . . . .

The decoding order of the pictures and receiving order of the picturesfrom the encoder to the decoder are also the same as order 1.

Here, inter-layer prediction may optionally be used.

That is, inter-layer prediction may not be applied to all pictures in alayer other than a top layer.

For example, in FIG. 3, when a target layer for display is spatialenhancement layer 1, all pictures in a spatial base layer may not needdecoding. In FIG. 3, for instance, pictures 1⁰ and 4⁰ may not needdecoding, because, as shown, two pictures 1⁰ and 4⁰ are not used asreference pictures in inter-layer prediction and not referred todirectly/indirectly by pictures used for inter-layer prediction amongpictures in the same spatial layer (base layer).

Thus, when 1⁰ and 4⁰ are not decoded, the delivery order of the picturesfrom the encoder to the decoder is order 1 but the decoding order of thepictures is order 2.

Order 2: 0⁰ 1⁰ 1¹ 2⁰ 2¹ 3⁰ 3¹ 4¹ 5⁰ 5¹ 6⁰ 6¹ . . . .

Comparing order 1 and order 2, pictures 1⁰ and 4⁰ are excluded fromdecoding in order 2. As described above, two pictures 1⁰ and 4⁰ are notused as reference pictures in inter-layer prediction and not referred todirectly/indirectly by pictures used for inter-layer prediction amongpictures in the same spatial layer (base layer).

As in order 2, when a picture neither used in inter-layer prediction norreferred to directly/indirectly by other pictures in the same layer usedfor inter-layer prediction is not decoded, the following advantages areobtained.

(1) Computing resource saving: computing resources are be used forcomputing/decoding of other pictures instead of computing/decoding ofsuch pictures, thereby increasing the speeds of encoding/decodingprocesses.

(2) Memory saving: such pictures are not computed, and thus it is notnecessary to allocate a memory space for the pictures, thereby saving amemory space.

However, although the advantages are obtained as a picture neither usedin inter-layer prediction nor referred to directly/indirectly by otherpictures in the same layer used for inter-layer prediction is notdecoded, there is needed a method of identifying which picture isskipped for decoding or which picture is removed from an input sequencewhen a target layer for decoding/display is given.

Hereinafter, a method of determining which picture is skipped/droppedfor decoding according to the present invention will be described.

Various methods, such as transmitting explicit information, changing adecoding method and memory management, may be used as a method for notdecoding a picture neither used in inter-layer prediction nor referredto directly/indirectly by other pictures in the same layer used forinter-layer prediction.

Embodiment I. Skipping of Decoding of Base Layer Picture Based onTransmission of Explicit Information

According to the present method, explicit information on each picture istransmitted, thereby indicating whether the picture is necessary forinter-layer prediction (referred to in inter-layer prediction of anotherlayer).

Transmitted first information may indicate whether a picture in a layerindicated by the first information is necessary for inter-layerprediction of a picture in another layer (referred to in inter-layerprediction of the picture in the other layer).

When the first information is defined as ilp_flag, ilp_flag equal to 1may indicate that a corresponding layer is necessary for inter-layerprediction of another layer, and ilp_flag equal to 0 may indicate thatthe layer is not necessary for inter-layer prediction of the otherlayer.

The first information may be transmitted in an NAL unit header or sliceheader. In addition, the first information may be transmitted within aconstraint. For example, when the first information is transmitted inthe slice header, according to a constraint, the value of the firstinformation may be the same in each slice header in the same picture.

Transmitted second information may be information indicating whether apicture specified by the second information is referred todirectly/indirectly by pictures needed for inter-layer prediction amongother pictures in the same layer.

For example, when the second information is defined as ilp_ref_flag,ilp_ref_flag equal to 1 may indicate that a picture specified byilp_ref_flag is a picture directly/indirectly referred to by picturesneeded for inter-layer prediction among other pictures in the samelayer.

The second information may be transmitted in an NAU unit header or sliceheader. In addition, the second information may be transmitted within acertain constraint. For example, when the second information istransmitted in the slice header, according to a constraint, the value ofthe second information may be the same in each slice header in the samepicture.

In the first information and the second information, a “picturespecified by information” refers to a picture to which the informationis applied.

The first information (for example, ilp_flag) and the second information(for example, ilp_ref_flag) may be independent of each other, and eachpicture may have different values of ilp_ref and ilp_ref_flag. Forexample, ilp_flag may have an independent value from that ofilp_ref_flag. Likewise, ilp_ref_flag may have an independent value fromthat of ilp_flag.

Hereinafter, for convenience of description, exemplary embodiments ofthe present invention will be described with reference to ilp_flag as anexample of first information and ilp_ref_flag as an example of secondinformation.

Embodiment I-1. Transmission of Information in NAL Unit Header

In transmission of embodiment I-1, the first information and the secondinformation may be transmitted in an NAL unit header. For instance,ilp_flag and ilp_ref_flag may be transmitted in the NAL unit header asin Table 1.

TABLE 1 nal_unit_header( ) { Descriptor   ...   ilp_flag u(1)  ilp_ref_flag u(1) ... }

As described above, ilp_flag indicates whether a current coded pictureis available for inter-layer prediction. ilp_ref_flag indicates that thecurrent coded picture is referred to or used by picture(s) needed forinter-layer prediction among pictures in the same layer.

That is, in the present embodiment, ilp_flag and ilp_ref_flag aretransmitted in the NAL unit header and indicate whether a picturecorresponding to the NAL unit header is directly/indirectly used forinter-layer prediction.

The decoder may receive ilp_flag and ilp_ref_flag in the NAL unitheader, and omit decoding of a target picture in a reference layer whenilp_flag and ilp_ref_flag indicate that the target picture is notdirectly/indirectly used for inter-layer prediction of a current block(a block to be decoded in the current layer).

Embodiment 1-2. Transmission of Information in Slice Segment Header

In transmission of embodiment 1-2, the first information and the secondinformation may be transmitted in a slice segment header. For instance,ilp_flag and ilp_ref_flag may be transmitted in the slice segment headeras in Table 2.

TABLE 2 slice_segment_header( ) { Descriptor   ...   ilp_flag u(1)  ilp_ref_flag u(1)   ... }

As described above, ilp_flag indicates whether a current slice isavailable for inter-layer prediction. ilp_ref_flag indicates that thecurrent slice is referred to or used by picture(s) needed forinter-layer prediction among pictures in the same layer.

That is, in the present embodiment, ilp_flag and ilp_ref_flag aretransmitted in the slice segment header and indicate whether a slicecorresponding to the slice segment header is directly/indirectly usedfor inter-layer prediction.

The decoder may receive ilp_flag and ilp_ref_flag in the slice segmentheader, and omit decoding of a target slice or picture in a referencelayer when ilp_flag and ilp_ref_flag indicate that the target slice isnot directly/indirectly used for inter-layer prediction of a currentblock (a block to be decoded in the current layer).

Alternatively, ilp_flag indicates whether a current coded picture isavailable for inter-layer prediction, in which the value of ilp_flag maybe the same in all slice segment headers of the coded picture. Also,ilp_ref_flag indicates whether the current coded pictures is referred toor used by picture(s) needed for inter-layer prediction among picturesin the same layer, in which the value of ilp_ref_flag may be the same inall slice segment headers of the coded picture.

In this case, the decoder may receive ilp_flag and ilp_ref_flag in theslice segment header, and omit decoding of a picture including a targetslice in a reference layer when ilp_flag and ilp_ref_flag indicate thatthe target slice is not directly/indirectly used for inter-layerprediction of a current block (a block to be decoded in the currentlayer).

Embodiment 1-3. Transmission of Information on Sub-Layers in VideoParameter Set (VPS)

In the present embodiment, the first information (ilp_flag) may betransmitted by two methods.

In method 1, the value of ilp_flag of each sub-layer may be the same forthe same sub-layer in a different layer as in Table 3.

TABLE 3 video_parameter_set_rbsp( ) { XDescriptor   ...  vps_max_sub_layers_minus1 u(3)   ...   for( i = 0; i<=vps_max_sub_layers_minus1; i++ ) {       ...     vps_ilp_flag[ i ] u(1)  }   ... }

In the VPS of Table 3, vps_max_sub_layers_minus1 specifies the maximumnumber of temporal sub-layers which may be present in a coded videosequence. In the VPS of Table 3, vps_ilp_flag[i] equal to 1 indicatesthat a picture/slice with a temporal ID (hereinafter, “temporalID”)equal to i among coded pictures/slices is available for inter-layerprediction, in which temporalID specifies a sub-layer. Otherwise, thatis, vps_ilp_flag[i] equal to 0 indicates that the picture/slice withtemporalID equal to i among the coded pictures/slices is unavailable forinter-layer prediction.

In method 2, the value of ilp_flag of each sub-layer may be differentfrom the value of ilp_flag of the same sub-layer in a different layer asin Table 4.

TABLE 4 video_parameter_set_rbsp( ) { Descriptor   ...  vps_max_num_layers_minus1 u(6)   vps_max_sub_layers_minus1 u(3)   ...  for( i = 0; i < vps_max_num_layers_minus1; i++ )     for( j = 0; j <=vps_max_sub_layers_minus1; j++ )      vps_ilp_flag[ i ][ j ] u(1)   }  ... }

In the VPS of Table 4, vps_max_num_layers_minus1 specifies the number oflayers that may be present in a coded video sequence, andvps_max_sub_layers_minus1 specifies the maximum number of temporalsub-layers that may be present in the coded video sequence. In Table 4,vps_ilp_flag[i][j] equal to 1 indicates that a picture/slice with alayer ID (hereinafter, “LayerID) equal to i and temporalID equal to jamong coded pictures/slices is available for inter-layer prediction, inwhich LayerID specifies a layer. Otherwise, that is, vps_ilp_flag[i]equal to 0 indicates that the picture/slice with LayerID equal to i andtemporalID equal to j among the coded pictures/slices is unavailable forinter-layer prediction.

Adoption of method 1 may decrease complexity and reduce traffic, andadoption of method 2 may obtain high flexibility for improvement ineffects of inter-layer prediction.

Embodiment 1-4. Combination of Transmissions of Inter-Layer PredictionInformation on Picture and Sub-Layers

According to the present embodiment, inter-layer information may betransmitted in combination. Specifically, ilp_flag and ilp_ref_flag in aslice segment header may be applied together with ilp_flag of asub-layer.

In this case, the following constraints may be applied. (1) Whenilp_flag is equal to 1 with respect to temporalID ilp_flag may be 1 or 0in a slice segment header with temporalID equal to i. (2) When ilp_flagis equal to 0 with respect to temporalID i, ilp_flag is 0 in a slicesegment header with temporalID equal to i.

Embodiment II. Skipping of Decoding of Base Layer Picture Based onDecoding Method

According to the present embodiment, a delivery/transport system betweenthe encoder and the decoder and the decoder may be improved inefficiency. The present embodiment allows the delivery system and thedecoder to effectively determine whether a coded picture with aparticular layer ID, ilp_flag, and ilp_ref_flag needs processing.

In detail, embodiment II allows the delivery/transport system betweenthe encoder and the decoder to determine whether a package including acoded picture with a particular layer ID, ilp_flag, and ilp_ref_flagneeds transmitting or dropping. Embodiment II also allows the decoder todetermine whether a coded picture with a particular layer ID, ilp_flag,and ilp_ref_flag needs coding.

In the present specification including embodiment II, TargetLayer, whichspecifies a target layer, is the layer ID of the highest layer displayedor output by the decoder. Also, LayerID is the layer ID of each codedpicture, ilp_flag represents the ilp_flag value of each coded picture,and ilp_ref_flag represents the ilp_ref_flag value of each codedpicture.

According to the present embodiment, when the following conditions aresatisfied with respect to particular TargetLayer, a coded picturecorresponding to TargetLayer is not coded or is dropped.

Condition (i) LayerID <TargetLayer

Condition (ii) ilp_flag value is false, that is, ilp_flag is equal to 0.

Condition (iii) ilp_ref_flag value is false, that is, ilp_ref_flag isequal to 0.

FIG. 4 is a diagram schematically illustrating the present embodiment,which shows an example of a method of removing coded pictures that donot need transmitting to the decoder or decoding.

Referring to FIG. 4, when a target layer is spatial enhancement layer 1,coded pictures 1⁰ and 4⁰ do not need transmitting to the decoder ordecoding, because pictures 1⁰ and 4⁰ satisfy the foregoing conditions(i) to (iii).

That is, in FIG. 4, pictures 1⁰ and 4⁰ are pictures in layer 0, whichhave LayerID smaller than that of TargetLayer and have ilp_flag andilp_ref_flag equal to 0.

Thus, in FIG. 4, encoding order/delivery order is order 3, and decodingorder is order 4.

Order 3: 0⁰ 0¹ 1⁰ 1¹ 2⁰ 2¹ 3⁰ 3¹ 4⁰ 4¹ 5⁰ 5¹ 6⁰ 6¹

Order 4: 0⁰ 0¹ 1¹ 2⁰ 2¹ 3⁰ 3¹ 4¹ 5⁰ 5¹ 6⁰ 6¹

Thus, according to embodiment II, the encoder may not transmit thepictures when all of conditions (i) to (iii) are satisfied, and thedecoder may drop or not decode the pictures when all of conditions (i)to (iii) are satisfied.

Embodiment III. Skipping of Decoding of Base Layer Picture Based onMemory Management

According to the present embodiment, it may be determined based onmemory management of the encoder and the decoder whether to store orretain a reconstructed picture of a coded picture corresponding to aparticular layer ID, ilp_flag, and ilp_ref_flag in a memory.

Also, according to the present embodiment, it may be determined based onmemory management of the encoder and the decoder whether to remove areconstructed picture of a coded picture corresponding to a particularlayer ID, ilp_flag, and ilp_ref_flag from a memory.

In embodiment III and the present specification, the memory may be adecoded picture buffer (DPB).

In the present embodiment, TargetLayer, which specifies a target layer,is also the layer ID of the highest layer displayed or decoded by thedecoder. Also, LayerID is the layer ID of each coded picture, ilp_flagrepresents the ilp_flag value of each coded picture, and ilp_ref_flagrepresents the ilp_ref_flag of each coded picture.

In the present embodiment, a reconstructed picture of a coded picturemay not be retained in the memory when the following conditions (i) to(iii) are satisfied with respect to given particular TargetLayer.

Condition (i) LayerID <TargetLayer

Condition (ii) ilp_flag value is false, that is, ilp_flag is equal to 0.

Condition (iii) ilp_ref_flag value is false, that is, ilp_ref_flag isequal to 0.

Also, in the present embodiment, with respect to given particularTargetLayer, after a coded picture with picture order count (POC), whichcorresponds to the output order of a picture, equal to pic_curr and withLayerID equal to curr_layer_id (that is, layer_id≤TargetLayer) isdecoded, a reference picture satisfying the following conditions (a) to(d) is removed among reference pictures in the memory.

Condition (a) POC of a reference picture is equal to poc_curr.

Condition (b) LayerID of a reference picture is smaller thancurr_layer_id.

Condition (c) ilp_flag of a reference picture is true, that is, ilp_flagof a reference picture, is equal to 1.

Condition (d) ilp_ref_flag of a reference picture is false, that is,ilp_ref_flag of a reference picture is equal to 0.

Here, condition (c) may be applied always or optionally.

According to the present embodiment, the decoder does not store orretain the reconstructed picture in the memory when LayerID <TargetLayerand ilp_flag and ilp_ref_flag are equal to 0.

In addition, after a picture with POC equal to pic_curr and LayerIDequal to curr_layer_id is decoded, the decoder may remove a picture withPOC equal to poc_curr, LayerID smaller than curr_layer_id, ilp_flagequal to 1, and ilp_ref_flag equal to 0 among reference pictures fromthe memory.

When memory management considering TargetLayer and utilizing ilp_flagand ilp_ref_flag according to embodiment III is employed, the size ofthe memory needed for processing and decoding a bitstream may be reduce.

FIG. 5 schematically illustrates an example of memory management.

Referring to FIG. 5, it may be verified that application of memorymanagement according to the present embodiment leads to a decrease inthe size of the memory.

In the example of FIG. 5, a picture structure based on a random accesssetting is illustrated. FIG. 5 shows two spatial scalability layersincluding four temporal levels.

In FIG. 5, pictures with temporal levels of 0 and 1 in a spatial baselayer may be used in inter-layer prediction. On the contrary, pictureswith temporal levels of 2 and 3 in the spatial base layer are not usedfor inter-layer prediction.

In this structure of FIG. 5, it is needed to allocate a memory forstoring at least six reference pictures in decoding the spatial baselayer. Here, when ilp_flag and ilp_ref_flag are not utilized, it isneeded to allocate a memory for storing at least 11 reference picturesin decoding a spatial enhancement layer.

FIG. 6 schematically illustrates an example of memory managementaccording to the present invention.

The example of FIG. 6 has the same picture structure as the example ofFIG. 5. However, unlike in FIG. 5, the example of FIG. 6 utilizes twopieces of information, ilp_flag and ilp_ref_flag.

When these two pieces of information, ilp_flag and ilp_ref_flag, areutilized, it may be indicated that the pictures with temporal levels of2 and 3 in the spatial base layer are not needed for decoding theenhancement layer based on inter-layer prediction and thus may beremoved.

Unlike in FIG. 5 where ilp_flag and ilp_ref_flag are not utilized, theexample of FIG. 6 utilizing ilp_flag and ilp_ref_flag, as shown, needsto allocate a memory for storing three pictures only in decoding thespatial base layer.

FIG. 7 schematically illustrates a case in which pictures not needed fordecoding are excluded in the example of FIG. 6.

As illustrated in FIG. 6, in the picture structure shown in FIGS. 5 to7, the pictures with temporal levels of 2 and 3 among the pictures inthe base layer are not used in decoding the enhancement layer and thusmay be removed from the memory as in FIG. 7. Thus, in FIGS. 6 and 7, amemory for storing three pictures is sufficient for the base layerunlike in FIG. 5.

FIG. 8 schematically illustrates an example of content/memory state/DPBstate in decoding each picture. In FIG. 8, each picture is describedaccording to decoding order and a picture structure is the same as thatin FIGS. 6 and 7.

It is assumed that TargetLayer is also 1 in FIG. 8.

Pictures marked with X are removed from the DPB.

As illustrated in FIG. 8, when memory management according to thepresent embodiment is applied, a necessary memory size is reduced from amemory size for storing 11 pictures to a memory size for storing 7pictures.

Embodiment IV. Skipping of Decoding of Base Layer Picture Based onConstruction and Management of Reference Picture List

In the present embodiment, a method of constructing a reference picturelist may be modified using ilp_flag and ilp_ref_flag.

When ilp_flag and ilp_ref_flag are used according to the presentembodiment, a reference picture list for a slice may be constructed asfollows.

{circle around (1)} When the ilp_flag value of the slice is 0, thereference picture list for the slice may be constructed based on areference picture set (RPS) only. That is, a current reference picturelist is not changed.

{circle around (2)} When the ilp_flag value of the slice is 1, thereference picture list for the slice is constructed by includingreference pictures which have ilp_ref_flag equal to 1 and are includedin the RPS for the slice.

An RPS refers to a set of reference pictures associated with a pictureand includes all reference pictures preceding the associated picture indecoding order. These reference pictures may be used for interprediction of the associated picture or a picture following theassociated picture in decoding order.

An RPS may include an RPS list. An RPS list may include a list ofshort-term reference pictures preceding a current picture in POC order,a list of short-term reference pictures following the current picture inPOC order, and a list of long-term reference pictures.

A reference picture list may be constructed by including at least one ofshort-term reference picture(s) preceding the current picture in POCorder, short-term reference picture(s) following the current picture inPOC order, long-term reference pictures, and inter-layer referencepicture(s).

A method of modifying a construction of a reference picture listaccording to embodiment IV will be described in detail.

In the present embodiment, NumRpsCurrTempList0 represents the number ofreference pictures for a current slice in temporal list 0.RefPicListTemp0 represents a list of POCs of reference pictures for acurrent slice in temporal list 0. NumRpsCurrTempList1 represents thenumber of reference pictures for a current slice in temporal list 1.RefPicListTemp1 represents a list of POCs of reference pictures for acurrent slice in temporal list 1. NumPocStCurrBefore represents thenumber of short-term reference pictures for a current slice whose POCsare smaller than the POC of the current slice. NumPocStCurrAfterrepresents the number of short-term reference pictures for a currentslice whose POCs are greater than the POC of the current slice.NumPocLtCurr represents the number of long-term reference pictures for acurrent slice. RefPicList0 represents list 0 L0 of reference picturesfor a current slice. RefPicList1 represents list 1 L1 of referencepictures for a current slice. ref_pic_list_modification_flag_10 isinformation indicating whether RefPicList0 needs modifying. Forinstance, ref_pic_list_modification_flag_10 equal to 1 indicates thatRefPicList0 needs modifying, and ref_pic_list_modification_flag_10 equalto 0 indicates that RefPicList0 does not need modifying.ref_pic_list_modification_flag_11 is information indicating whetherRefPicList1 needs modifying. For instance,ref_pic_list_modification_flag_11 equal to 1 indicates that RefPicList1needs modifying, and ref_pic_list_modification_flag_11 equal to 0indicates that RefPicList1 does not need modifying. IlpRefList is a listof ilp_ref_flag of pictures. IlpRefList [x] is ilp_ref_flag of picturex, and index x may represent the POC of a picture.

A reference picture list according to the present embodiment isconstructed as follows.

At the beginning of the decoding process of each slice, a referencepicture list RefPicList0 and RefPicList1 for a B picture may beconstructed as follows.

A variable NumRpsCurrTempList0 may be set toMax(num_ref_idx_10_active_minus1+1, NumPocTotalCurr), and a listRefPicListTemp0 may be constructed as in Table 5.

TABLE 5 rIdx = 0 while( rIdx < NumRpsCurrTempList0 ) {  for( i = 0; i <NumPocStCurrBefore && rIdx < NumRpsCurrTempList0; i++ )     if (ilp_flag== 0 || (ilp_flag == 1 && IlpRefList[RefPicSetStCurrBefore[ i ]]))       RefPicListTemp0[ rIdx++ ] = RefPicSetStCurrBefore[ i ]  for( i =0; i < NumPocStCurrAfter && rIdx < NumRpsCurrTempList0; i++ )     if(ilp_flag == 0 || (ilp_flag == 1 && IlpRefList[RefPicSetStCurrAfter [ i]]))        RefPicListTemp0[ rIdx++ ] = RefPicSetStCurrAfter[ i ]  for(i = 0; i < NumPocLtCurr && rIdx < NumRpsCurrTempList0; i++ )     if(ilp_flag == 0 || (ilp_flag == 1 && IlpRefList[RefPicSetLtCurr [ i ]]))       RefPicListTemp0[ rIdx++ ] = RefPicSetLtCurr[ i ] }

Thus, a reference picture list RefPicList0 may be constructed as inTable 6.

TABLE 6 for( rIdx = 0; rIdx <= num_ref_idx_l0_active_minus1; rIdx++)   RefPicList0[ rIdx ] = ref_pic_list_modification_flag_l0 ?   RefPicListTemp0[ list_entry_l0[ rIdx ] ] : RefPicListTemp0[ rIdx ]

Meanwhile, when a slice is a B slice, the second reference picture listRefPicList1 may be used. In this case, a variable NumRpsCurrTempList1may be set to Max(num_ref_idx_10_active_minus1+1, NumPocTotalCurr), anda list RefPicListTemp1 may be constructed as in Table V.

TABLE 7 rIdx = 0 while( rIdx < NumRpsCurrTempList1 ) {    for( i = 0; i< NumPocStCurrAfter && rIdx < NumRpsCurrTempList1; rIdx++, i++ )      if (ilp_flag == 0 || (ilp_flag == 1 &&IlpRefList[RefPicSetStCurrAfter [ i ]]))          RefPicListTemp1[ rIdx] = RefPicSetStCurrAfter[ i ]    for( i = 0; i < NumPocStCurrBefore &&rIdx < NumRpsCurrTempList1; rIdx++, i++ )       if (ilp_flag == 0 ||(ilp_flag == 1 && IlpRefList[RefPicSetStCurrBefore [ i ]]))         RefPicListTemp1[ rIdx ] = RefPicSetStCurrBefore[ i ]    for( i= 0; i < NumPocLtCurr && rIdx < NumRpsCurrTempList1; rIdx++, i++ )      if (ilp_flag == 0 || (ilp_flag == 1 && IlpRefList[RefPicSetLtCurr[ i ]]))          RefPicListTemp1[ rIdx ] = RefPicSetLtCurr[ i ] }

Thus, when the slice is a B slice, a reference picture list RefPicList1may be constructed as in Table 8.

TABLE 8 for( rIdx = 0; rIdx <= num_ref_idx_l1_active_minus1; rIdx++)   RefPicList1[ rIdx ] = ref_pic_list_modification_flag_l1 ?   RefPicListTempl[ list_entry_l1[ rIdx ] ]: RefPicListTemp1[ rIdx ]

Referring to Tables 5 to 8, the temporal lists (RefPicListTemp0,RefPicListTemp1) are constructed according to {circle around (1)} and{circle around (2)} using ref_flag and ilp_ref_flag. Then, when there isno modification in the reference picture lists(ref_pic_list_modification_flag_lX=0, X=0 or 1), the temporal lists maybe the reference picture lists. When there is a modification in thereference picture lists (ref_pic_list_modification_flag_lX=1), thetemporal lists modified based on list entry information(list_entry_lx[rIdx], lx=10 or 11) transmitted from the encoder may bethe reference picture lists.

The methods of skipping decoding of a reference picture in a base layerbased on signaling (information transmission), a decoding method, memorymanagement, and construction of a reference picture list according tothe present invention have been described.

In addition to the foregoing methods, a method of controllinginter-layer prediction in a picture level may be considered in order toperform efficient inter-layer prediction.

In a multilayer bitstream, decoding complexity may be reduced when anon-highest picture or a non-outputted picture is not needed forinter-layer prediction, because these pictures may not be decoded or beremoved as described above.

Hereinafter, a method of controlling inter-layer prediction in a picturelevel will be described based on a signaling method and a decodingmethod.

Embodiment V. Control of Inter-Layer Prediction Based on SignalingMethod Embodiment V-1

In embodiment V-1, ilp_flag and ilp_ref_flag may be transmitted in aslice segment header.

TABLE 9 slice_segment_header( ) { Descriptor   ...   ilp_flag u(1)  ilp_ref_flag u(1)   ... }

As described above, ilp_flag indicates whether a current slice isavailable for inter-layer prediction. ilp_ref_flag indicates whether thecurrent slice is referred to or used by picture(s) needed forinter-layer prediction among pictures in the same layer.

That is, in the present embodiment, ilp_flag and ilp_ref_flag aretransmitted in the slice segment header and indicates whether a slicecorresponding to the slice segment header is directly/indirectly usedfor inter-layer prediction.

The decoder receives ilp_flag and ilp_ref_flag in the slice segmentheader, and may skip decoding of a target slice in a reference layer ora picture when ilp_flag and ilp_ref_flag indicate that the target sliceis not directly/indirectly used in inter-layer prediction of a currentblock (block to be decoded in a current layer).

Here, the values of ilp_flag and ilp_ref_flag may be controlled in apicture level.

In detail, ilp_flag indicates whether a current coded picture isavailable for inter-layer prediction, and the value of ilp_flag may bethe same in all slice segment headers of the coded picture.

Further, ilp_ref_flag indicates whether the current coded picture isreferred to or used by picture(s) needed for inter-layer predictionamong pictures in the same layer, and the value of ilp_ref_flag may bethe same in all slice segment headers of the coded picture.

In this case, the decoder receives ilp_flag and ilp_ref_flag in theslice segment header, and may skip decoding of a picture including atarget slice in a reference layer when ilp_flag and ilp_ref_flagindicate that the target slice is not directly/indirectly used ininter-layer prediction of the current block (block to be decoded in thecurrent layer).

Embodiment V-2

In embodiment V-2, information on inter-layer prediction may be signaledin a slice segment header as in Table 10.

TABLE 10 slice_segment_header( ) { Descriptor   ...   need_for_ilp_flagu(1)   ... }

In Table 10, need_for_ilp_flag indicates whether a current slice is usedfor inter-layer prediction, or is used or referred to in interprediction of other picture(s) needed for inter-layer prediction amongpictures in the same layer. That is, when need_for_ilp_flag is equal to1, a picture indicated by need_for_ilp_flag is directly/indirectlyused/referred to in inter prediction. When need_for_ilp_flag is equal to0, the picture indicated by need_for_ilp_flag is not directly/indirectlyused/referred to in inter-layer prediction.

Here, need_for_ilp_flag may further be controlled in a picture level.

In this case, as described above, need_for_ilp_flag indicates whetherthe current slice is used for inter-layer prediction, or is used orreferred to in inter prediction of other picture(s) needed forinter-layer prediction among pictures in the same layer. Here, the valueof need_for_ilp_flag is the same in all slice segment headers of a codedpicture.

Embodiment VI. Control of Inter-Layer Prediction Based On Effect ofDecoding Process Embodiment VI-1

In embodiment VI-1, TargetDecLayerIdList is a list of nuh_layer_idvalues, in which the nuh_layer_id values of NAL units to be decoded arearranged in ascending order. nuh_layer_id is information on an NAL unitlevel specifying a layer in a multilayer structure.TargetOutputLayerIdList is a list of nuh_layer_id values, in which thenuh_layer_id values of NAL units to be output are arranged in ascendingorder. HighestTid represents the highest temporal sub-layer to bedecoded.

In an example of embodiment VI-1, when the ilp_flag and ilp_ref_flagvalues of a current picture are false and nu_layer_id of a current NALunit is not included in TargetOutputLayerIdList, the current picture maybe skipped in a decoding process. Thus, the decoder may not decode thecurrent picture when the ilp_flag and ilp_ref_flag values of the currentpicture are 0 and nuh_layer_id of the current NAL unit is not includedin TargetOutputLayerIdList.

In another example of embodiment VI-1, when the ilp_flag andilp_ref_flag values of a current slice are false and nu_layer_id of acurrent NAL unit is not included in TargetOutputLayerIdList, the currentslice may be skipped in a decoding process. Thus, the decoder may notdecode the current slice when the ilp_flag and ilp_ref_flag values ofthe current slice are 0 and nuh_layer_id of the current NAL unit is notincluded in TargetOutputLayerIdList.

In still another example of embodiment VI-1, when the ilp_flag andilp_ref_flag values of a current picture are false and nu_layer_id of acurrent NAL unit is not included in TargetOutputLayerIdList, thePicOutputFlag value of the current picture may be set to 0. ThePicOutputFlag value of the current picture may indicate whether tooutput the current picture. For example, PicOutputFlag equal to 1 mayindicate that the picture is output, and PicOutputFlag equal to 0 mayindicate that the picture is not output. Thus, the decoder sets thePicOutputFlag value of the current picture to 0 and may not output thecurrent picture when the ilp_flag and ilp_ref_flag values of the currentpicture are 0 and nuh_layer_id of the current NAL unit is not includedin TargetOutputLayerIdList.

In yet another example of embodiment VI-1, when the ilp_flag andilp_ref_flag values of a current slice are false and nu_layer_id of acurrent NAL unit is not included in TargetOutputLayerIdList, thePicOutputFlag value of the current picture may be set to 0. Thus, thedecoder sets the PicOutputFlag value of the current picture to 0 and maynot output the current picture when the ilp_flag and ilp_ref_flag valuesof the current slice are 0 and nuh_layer_id of the current NAL unit isnot included in TargetOutputLayerIdList.

Embodiment VI-2

In embodiment VI-2, TargetDecLayerIdList is a list of nuh_layer_idvalues, in which the nuh_layer_id values of NAL units to be decoded arearranged in ascending order. nuh_layer_id is information on an NAL unitheader level specifying a layer in a multilayer structure.TargetOutputLayerIdList is a list of nuh_layer_id values, in which thenuh_layer_id values of NAL units to be output are arranged in ascendingorder. HighestTid represents the highest temporal sub-layer to bedecoded.

In an example of embodiment VI-2, when the need_for_ilp_flag value of acurrent picture is false and nuh_layer_id of a current NAL unit is notincluded in TargetOutputLayerIdList, the current picture may be skippedin a decoding process. Thus, the decoder may not decode the currentpicture when the need_for_ilp_flag value of the current picture is 0 andnuh_layer_id of the current NAL unit is not included inTargetOutputLayerIdList.

In another example of embodiment VI-2, when the need_for_ilp_flag valueof a current picture is false and nuh_layer_id of a current NAL unit isnot included in TargetOutputLayerIdList, a current slice may be skippedin a decoding process. Thus, the decoder may not decode the currentslice when the need_for_ilp_flag value of the current picture is 0 andnuh_layer_id of the current NAL unit is not included inTargetOutputLayerIdList.

In still another example of embodiment VI-2, when the need_for_ilp_flagvalue of a current picture is false and nuh_layer_id of a current NALunit is not included in TargetOutputLayerIdList, PicOutputFlag of thecurrent picture may be set to 0. The PicOutputFlag value of the currentpicture may indicate whether to output the current picture. For example,PicOutputFlag equal to 1 may indicate that the picture is output, andPicOutputFlag equal to 0 may indicate that the picture is not output.Thus, the decoder sets the PicOutputFlag value of the current picture to0 and may not output the current picture when the need_for_ilp_flagvalue of the current picture is 0 and nuh_layer_id of the current NALunit is not included in TargetOutputLayerIdList.

In yet another example of embodiment VI-2, when the need_for_ilp_flagvalue of a current slice is false and nuh_layer_id of a current NAL unitis not included in TargetOutputLayerIdList, PicOutputFlag of a currentpicture may be set to 0. Thus, the decoder sets the PicOutputFlag valueof the current picture to 0 and may not output the current picture whenthe need_for_ilp_flag value of the current slice is 0 and nuh_layer_idof the current NAL unit is not included in TargetOutputLayerIdList.

The method of controlling decoding of a reference layer through aconstruction of a reference picture list has been illustrated above, inwhich the reference picture list may be managed in view of a pictureunnecessary for inter-layer prediction.

Hereinafter, a method of managing a reference picture list according tothe present invention will be described in detail.

Embodiment VII. Management of Reference Picture List in View of PicturesUnnecessary for Inter-Layer Prediction

As described above, in a multilayer bitstream, decoding complexity maybe reduced when a non-highest picture or a non-outputted picture is notneeded for inter-layer prediction, because these pictures may not bedecoded or be removed as described above.

The present embodiment illustrates a method of removing a pictureunnecessary for prediction of a current picture (current block) ormarking the unnecessary picture as “unused for reference” amonginter-layer reference pictures included in a reference picture list indetail.

Embodiment VII-1

When information indicating use of inter-layer prediction is transmittedonly in a sequence level, Table 11 may be applied.

TABLE 11 vps_extension( ) { Descriptor  ...  for( i = 1; i<=vps_max_layers_minus1; i++ ) {   // layer dependency   for( j = 0; j <i;j++ )    direct_dependency_flag[ i ][ j ] u(1)  }  for( i = 0; i <=vps_max_layers_minus1; i++ ) {   max_sublayer_for_ilp_plus1 [ i ] u(3) }

In Table 11, direct_dependency_flag[i][j] specifies dependency betweenlayers in a multilayer structure. When direct_dependency_flag[i][j] isequal to 0, a layer with an index of j is not a direct reference layerfor prediction of a layer with an index of i. Whendirect_dependency_flag[i][j] is equal to 1, the layer with an index of jmay be a direct reference layer for prediction of the layer with anindex of i.

In Table 11, max_sublayer_for_ilp_plus1[i] specifies the maximumsub-layer in layer i which may be used/referred to in inter-layerprediction. When max_sublayer_for_ilp_plus1[i] is equal to 0, only arandom access point (RAP) picture may be used for inter-layerprediction. When max_sublayer_for_ilp_plus1[i] is greater than 0,pictures with temporalID smaller than or equal tomax_sublayer_for_ilp_plus1[i]−1 may be used for inter-layer prediction.

In this case, an inter-layer reference picture in a reference layer withtemporalID greater than or equal to max_sublayer_for_ilp_plus1[layer IDof reference layer] is not included in a reference picture set for acurrent picture. That is, in the present embodiment, a picture in areference layer with temporalID greater than or equal to temporalID ofthe maximum sub-layer specified by max_sublayer_for_ilp_plus1[layered ofreference layer] is not included in a reference picture list for thecurrent picture.

To this end, a process of reconstructing an inter-layer referencepicture set may be modified as follows.

Output from the process of reconstructing the inter-layer referencepicture set may be an updated list of inter-layer reference pictures,RefPicSetInterLayer. The list RefPicSetInterLayer is set to be empty atfirst and may be derived as in Table 12 through the reconstructingprocess.

TABLE 12 for(i=0;i <NumDirectRefLayers[LayerIdInVps[nuh_layer_id]];i++){ RefPicSetInterLayer[ i ] = the picture with picture order count equalto PicOrderCnt and nuh_layer_id equal to RefLayerId[LayerIdInVps[nuh_layer_id ]] [ i ] but whose TemproalId is not greater than or equalto max_sublayer_for_lip_plus1[RefLayerId[ LayerIdInVps [ nuh_layer_id]][ i ]]  RefPicSetInterLayer[ i ] is marked as “used for long-termreference” }

After a picture in an upper layer is decoded, picture(s) in the DPBhaving a lower layer ID than the upper layer and the same POC as theupper layer may be marked as “unused for reference” or removed from theDPB when satisfying the following condition.

<Condition> temporalID of a picture is greater thanmax_sublayer_for_ilp_plus1[refLayerId [LayerIdInVps[Nuh_layer_id]][i]].

The foregoing condition may also be applied as follows: temporalID of apicture is greater than temporalID of the maximum sub-layer specified bymax_sublayer_for_ilp_plus1.

In this case, the decoder may remove the picture from the DPB or marksthe picture as “unused for reference” so that the picture is notreferred to in inter-layer prediction. The picture removed from the DPBor marked as “unused for reference” is not included in a referencepicture list used for inter-layer prediction.

Embodiment VII-2

In an example of various methods for indicating whether a picture isneeded for inter-layer prediction, two pieces of information, forexample, ilp_flag and ilp_ref_flag, may be used.

Table 13 illustrates an example of signaling ilp_flag and ilp_ref_flagin a slice segment header.

TABLE 13 slice_segment_header( ) { Descriptor ...   ilp_flag u(1)  ilp_ref_flag u(1)   ... }

In Table 13, ilp_flag indicates whether a current coded picture isavailable for inter-layer prediction. ilp_ref_flag indicates whether thecurrent coded picture is referred to or used by picture(s) needed forinter-layer prediction among pictures in the same layer.

Here, when ilp_flag is equal to 0, a picture indicated by ilp_flag isnot referred to/used in inter-layer prediction and thus is not includedin a list of inter-layer reference pictures for the current picture.

To this end, a process of reconstructing an inter-layer referencepicture set may be modified as follows.

Output from the process of reconstructing the inter-layer referencepicture set may be an updated list of inter-layer reference pictures,RefPicSetInterLayer. The list RefPicSetInterLayer is set to be empty atfirst and may be derived as in Table 14 through the reconstructingprocess.

TABLE 14 for( i = 0; i < NumDirectRefLayers[ LayerIdInVps[ nuh_layer_id] ]; i++ ) {  RefPicSetInterLayer[ i ] = the picture with picture ordercount equal to PicOrderCnt and nuh_layer_id equal to RefLayerId[LayerIdInVps[ nuh_layer_id ]][ i ] and whose ilp_flag is equal to 1 RefPicSetInterLayer[ i ] is marked as “used for long-term reference” }

When the two pieces of information are used to indicate inter-layerprediction, after a picture in an upper layer is decoded, picture(s) inthe DPB having a lower layer ID than the upper layer and the same POC asthe upper layer may be marked as “unused for reference” or removed fromthe DPB when satisfying the following condition.

<Condition> ilp_flag is equal to 0 and ilp_ref_flag is equal to 0, orilp_flag equal to 1 and ilp_ref_flag is equal to 0.

Thus, when ilp_ref_flag is equal to 0, the decoder may mark the pictureindicated by ilp_ref_flag as “unused for reference” or remove thepicture from the DPB so that the picture is not referred to/used ininter-layer prediction. The picture removed from the DPB or marked as“unused for reference” is not included in a reference picture list usedfor inter-layer prediction.

Meanwhile, only one piece of information (need_for_ilp_flag) may beused, instead of two pieces of information (ilp_flag and ilp_ref_flag),to indicate whether a picture is necessary for inter-layer prediction.

Table 15 illustrates an example of signaling need_for_ilp_flag in aslice segment header.

TABLE 15 slice_segment_header( ) { Descriptor   ...   need_for_ilp_flagu(1)   ... }

In Table 15, need_for_ilp_flag indicates whether a current slice is usedfor inter-layer prediction or is referred to or used in interlayerprediction of other picture(s) needed for inter-layer prediction amongpictures in the same layer. That is, when need_for_ilp_flag is equal to1, a picture indicated by need_for_ilp_flag is directly/indirectlyused/referred to in inter-layer prediction. When need_for_ilp_flag isequal to 0, the picture indicated by need_for_ilp_flag is notdirectly/indirectly used/referred to in inter-layer prediction.

Thus, when need_for_ilp_flag is equal to 0, the picture indicated byneed_for_ilp_flag is not referred to/used in inter-layer prediction andthus is not included in a list of inter-layer reference pictures for thecurrent picture.

To this end, a process of reconstructing an inter-layer referencepicture set may be modified as follows.

Output from the process of reconstructing the inter-layer referencepicture set may be an updated list of inter-layer reference pictures,RefPicSetInterLayer. The list RefPicSetInterLayer is set to be empty atfirst and may be derived as in Table 16 through the reconstructingprocess.

TABLE 16 for( i = 0; i < NumDirectRefLayers[ LayerIdInVps[ nuh_layer_id] ]; i++ ) {  RefPicSetInterLayer[ i ] = the picture with picture ordercount equal to PicOrderCnt and nuh_layer_id equal to RefLayerId[LayerIdInVps[ nuh_layer_id ]][ i ] and whose need_for_ilp_flag is equalto 1  RefPicSetInterLayer[ i ] is marked as “used for long-termreference” }

Thus, when one piece of information (need_for_ilp_flag) is used toindicate inter-layer prediction, after a picture in an upper layer isdecoded, picture(s) in the DPB having a lower layer ID than the upperlayer and the same POC as the upper layer may be marked as “unused forreference” or removed from the DPB when satisfying the followingcondition.

<Condition> need_for_ilp_flag is equal to 0

Thus, when need_for_ilp_flag is equal to 0, the decoder may mark thepicture indicated by need_for_ilp_flag as “unused for reference” orremove the picture from the DPB so that the picture is not referredto/used in inter-layer prediction. The picture removed from the DPB ormarked as “unused for reference” is not included in a reference picturelist used for inter-layer prediction.

Embodiment VII-3

In embodiment VII-3, both signaling in a sequence level and signaling ina picture level may be used.

In an example of embodiment VII-3, a process of reconstructing aninter-layer reference picture set may be modified as follows.

Output from the process of reconstructing the inter-layer referencepicture set may be an updated list of inter-layer reference pictures,RefPicSetInterLayer. The list RefPicSetInterLayer is set to be empty atfirst and may be derived as in Table 17 through the reconstructingprocess.

TABLE 17 for( i = 0; i < NumDirectRefLayers[ LayerIdInVps[ nuh_layer_id] ]; i++ ) {  RefPicSetInterLayer[ i ] = the picture with picture ordercount equal to PicOrderCnt and nuh_layer_id equal to RefLayerId[LayerIdInVps[ nuh_layer_id ]][ i ] and whose ilp_flag is equal to 1 butwhose TemporalId is not greater than or equal tomax_sublayer_for_ilp_plus1[RefLayerId[ LayerIdInVps [ nuh_layer_id ]][ i]]  RefPicSetInterLayer[ i ] is marked as “used for long-term reference”}

In an example where Table 17 is applied, after a picture in an upperlayer is decoded, picture(s) in the DPB having a lower layer ID than theupper layer and the same POC as the upper layer may be marked as “unusedfor reference” or removed from the DPB when satisfying the followingcondition.

<Condition>(1) temporalID of a picture is greater thanmax_sublayer_for_ilp_plus1[refLayerId [LayerIdInVps[Nuh_layer_id]][i]],(2) ilp_flag is equal to 0 and ilp_ref_flag is equal to 0, or (3)ilp_flag is equal to 1 and ilp_ref_flag is equal to 0. Here, (1) may bereplaced by a condition that temporalID of a picture is greater thantemporalID of the maximum sub-layer specified bymax_sublayer_for_ilp_plus1, (2) and (3) may be replaced by a conditionthat need_for_ilp_flag is equal to 0.

According to another example of embodiment VII-3, Table 17 illustratingthe process of reconstructing the inter-layer reference picture set maybe replaced by Table 18.

Output from the process of reconstructing the inter-layer referencepicture set may be an updated list of inter-layer reference pictures,RefPicSetInterLayer. The list RefPicSetInterLayer is set to be empty atfirst and may be derived as in Table 18 through the reconstructingprocess.

TABLE 18 for( i = 0; i < NumDirectRefLayers[ LayerIdInVps[ nuh_layer_id] ]; i++ ) {  RefPicSetInterLayer[ i ] = the picture with picture ordercount equal to PicOrderCnt and nuh_layer_id equal to RefLayerId[LayerIdInVps[ nuh_layer_id ]][ i ] and whose ilp_flag is equal to 1 butwhose TemporalId is not greater than or equal tomax_sublayer_for_ilp_plus1[RefLayerId[ LayerIdInVps [ nuh_layer_id ]][ i]]  RefPicSetInterLayer[ i ] is marked as “used for long-term reference”}

FIG. 9 is a flowchart schematically illustrating an example of anoperation of a video encoder according to the present invention.

Referring to FIG. 9, the encoder may decode and store a picture in areference layer (S910). In scalable video coding which supports amultilayer structure, the encoder may decode the picture in thereference layer and store the decoded picture in a decoded picturebuffer (DPB). The decoded picture in the reference layer may be apicture belonging to the same AU as a picture to be decoded in a currentlayer.

When the picture in the reference layer is neither referred to inprediction of a current block nor referred to in prediction of anotherpicture in the reference layer which is referred to in prediction of thecurrent block, the encoder may mark the picture in the reference layeras unused for reference.

In this case, the encoder may transmit information on the pictureneither referred to in prediction of the current block nor referred toin prediction of the other picture in the reference layer which isreferred to in prediction of the current block to a decoder throughreference information (for example, ilp_flag, ilp_ref_flag,need_for_ilp_flag emd), which will be described.

The encoder may decode or store the picture not marked as unused forreference. Alternatively, the encoder may remove the picture marked asunused for reference from a memory.

The encoder may derive an inter-layer reference picture for predictionof the current block (S920). The encoder may derive a reference picture(inter-layer reference picture) for a block to be decoded (currentblock) of a current picture in the current layer from the picture in thereference layer. In this case, the encoder may derive the inter-layerreference picture by performing sampling in view of a phase andresolution.

The encoder may derive the inter-layer reference picture based onpictures which are available for reference in prediction of the currentblock or available for reference in prediction of another picture in thereference layer referred to in prediction of the current block.

Here, the information specifying the picture neither referred to inprediction of the current block nor referred to in prediction of anotherpicture in the reference layer which is referred to in prediction of thecurrent block may be transmitted to the decoder through the referenceinformation (for example, ilp_flag, ilp_ref_flag, need_for_ilp_flagemd), which will be described.

The encoder may construct a reference picture list used for predictionof the current block (S930). The reference picture list may be referencepicture list L0, or reference picture list L0 and reference picture listL1 depending on a current slice (slice to be decoded).

The reference picture list may include pictures in the same layer(current layer) as the current picture and a reference picture(inter-layer reference picture) derived from a reference layer differentfrom the current layer. Here, the reference picture list may include asingle inter-layer reference picture derived from a single referencelayer or two or more inter-layer reference pictures derived from two ormore reference layers.

The encoder may derive a predicted sample of the current block based onthe reference picture list (S940). The encoder may perform interprediction on the current block using at least one of the pictures inthe reference picture list. For instance, the encoder may predict thecurrent block using the inter-layer reference picture derived from thereference layer.

The encoder may derive a reconstructed sample based on the predictedsample (S950). The encoder may derive the reconstructed sample by addingthe predicted sample and a residual signal. The encoder may derive thereconstructed sample by the current block or current picture.

The encoder may transmit the reference information (S960). The encodermay transmit information on a picture in the reference layer availablefor inter-layer prediction through the reference information. Forinstance, when a picture in the reference layer is available forreference in prediction of the current block or available for referencein prediction of another picture in the reference layer which isreferred to in prediction of the current block, the referenceinformation may indicate that the picture in the reference layer isavailable for reference in inter-layer prediction. Alternatively, when apicture in the reference layer is neither referred to in prediction ofthe current block nor referred to in prediction of another picture inthe reference layer which is referred to in prediction of the currentblock, the reference information may indicate that the picture in thereference layer is not referred to in inter-layer prediction.

The encoder may transmit the reference information for the currentpicture in a slice segment header. Alternatively, the encoder maytransmit the reference information for the current picture in a picturelevel.

In transmitting information, the encoder may transmit residualinformation that is a difference between an original signal and apredicted signal in addition to the reference information.

Although encoding including prediction of the current block in themultilayer structure has been described, the present invention is notlimited thereto. Details of encoding may include all technical featuresdescribed in the present specification.

FIG. 10 is a flowchart schematically illustrating an example of anoperation of a video decoder according to the present invention.

Referring to FIG. 10, the decoder may receive reference information(S1010). The decoder may receive, from an encoder, the referenceinformation indicating whether a picture in a reference layer isavailable for inter-layer prediction.

For example, when the picture in the reference layer is available forreference in prediction of a current block or available for reference inprediction of another picture in the reference layer which is referredto in prediction of the current block, the reference information mayindicate that the picture in the reference layer is available forinter-layer prediction.

Alternatively, when the picture in the reference layer is neitherreferred to in prediction of the current block nor referred to inprediction of another picture in the reference layer which is referredto in prediction of the current block, the reference information mayindicate that the picture in the reference layer is not used ininter-layer prediction.

The decoder may decode and store the picture in the reference layer(S1020). The decoder may mark the picture, indicated to be unavailablefor inter-layer prediction by the reference information, as “unused forreference.”

The decoder may decode or store the picture not marked as unused forreference. Alternatively, the decoder may remove the picture marked asunused for reference from a memory (for example, DPB).

The decoder may derive an inter-layer reference picture referred to inprediction of the current block (S1030). The decoder may derive theinter-layer reference picture referred to in prediction of the currentblock in a current layer from the decoded picture in the referencelayer.

For example, the decoder may derive the inter-layer reference picturebased on pictures indicated by the reference information to be availablefor inter-layer prediction among pictures in the reference layer. Here,the decoder may derive a single inter-layer reference picture from asingle reference layer or a plurality of inter-layer reference picturesderived from a plurality of reference layers.

Here, when the picture, indicated by the reference information to beunavailable for inter-layer prediction, is marked as “unused forreference” in operation S1020, the decoder may derive the inter-layerreference picture from the picture in the reference layer which is notmarked as unused for reference.

In addition, when the picture, indicated by the reference information tobe unavailable for inter-layer prediction, is marked as “unused forreference” in operation S1020 and the marked picture is removed from thememory, the decoder may derive the inter-layer reference picture from apicture in the reference layer which remains in the memory.

Furthermore, the decoder may derive an inter-layer reference picturefrom the picture indicated by the reference information to be availablefor inter-layer prediction in the layer indicated to be available forinter-layer prediction by the current picture.

Here, the decoder may derive the inter-layer reference picture from apicture belonging to the same AU as the current picture among thepictures in the reference layer.

The decoder may derive a reference picture list to be used for interprediction of the current block (S1040). The decoder may construct areference picture list including the inter-layer reference picture and areference picture in the current layer.

The reference picture list may be reference picture list L0, orreference picture list L0 and reference picture list L1 depending on acurrent slice (slice to be decoded).

The reference picture list may include pictures in the same layer(current layer) as the current picture and a reference picture(inter-layer reference picture) derived from a reference layer differentfrom the current layer. Here, the reference picture list may include asingle inter-layer reference picture derived from a single referencelayer or two or more inter-layer reference pictures derived from two ormore reference layers.

The decoder may derive a predicted sample of the current block (S1050).The decoder may derive the predicted sample of the current block bypredicting the current block in the current layer based on the referencepicture list.

The decoder may derive a reconstructed sample based on the predictedsample (S1060). The decoder may derive the reconstructed sample byadding the predicted sample and a residual signal. The decoder mayderive the reconstructed sample by the current block unit or currentpicture unit.

In the present specification, the terms “base layer” and “spatial baselayer” are used in a mixed manner and the terms “enhancement layer” and“spatial enhancement layer” are used in a mixed manner, which is forconvenience of description. The embodiments of the present invention arenot limited to the spatial base layer and the spatial enhancement layer.A spatial base layer is used as an example of a case where resolution isapplied as scalability of a base layer, and a spatial enhancement layeris used as an example of a case where resolution is applied asscalability of an enhancement layer. It should be noted that theembodiments of the present invention may be applied in the same mannerto various scalabilities illustrated above (view, bit rate, frame rate,or the like) in addition to resolution.

In the present specification, the expressions “referred to ininter(-layer) prediction,” “used in or for inter(-layer) prediction,”and “needed for inter(-layer) prediction” are used in a mixed manner forconvenience of description. These expressions are neither contradictorynor exclusive expressions but may refer to the same meaning. Inaddition, a relationship of inclusion may be established between theseexpressions. In detail, the expressions “being used in or forinter-layer prediction” and “being needed for inter-layer prediction”may include the expression “being referred to in prediction of a directreference picture for prediction of a picture (block) to be predicted.”

Generality of NAL unit header and slice segment header

Use: Including indirect reference

While the methods in the above-mentioned exemplary system have beendescribed on the basis of flowcharts including a series of steps orblocks, the invention is not limited to the order of steps and a certainstep may be performed in a step or an order other than described aboveor at the same time as described above. Since the above-mentionedembodiments may include various examples, combinations of embodimentsmay also be understood as an embodiment. Therefore, the inventionincludes all substitutions, corrections, and modifications belonging tothe appended claims.

1-19. (canceled)
 20. A video decoding method by a video decoder,comprising: receiving flag information for whether a picture in areference layer is not used for an inter-layer prediction; decoding andstoring pictures in the reference layer; deriving an inter-layerreference picture for a current block from at least one of the decodedpictures in the reference layer based on the flag information;constructing a reference picture list comprising the inter-layerreference picture in the reference layer and a reference picture in thecurrent layer; deriving a predicted sample of the current block in thecurrent layer based on the inter-layer reference picture comprised inthe reference picture list; deriving a reconstructed picture based onthe predicted sample and a residual sample of the current block; andperforming deblocking filtering on the reconstructed picture, whereinwhen the flag information indicates that a specific picture in thereference layer is not used for the inter-layer prediction, the specificpicture is not comprised in the inter-layer reference picture set, andwherein the flag information is received through a slice segment header,and in all slice segment header of the picture in the reference layer, avalue of the flag information is set to the same.
 21. The method ofclaim 20, wherein when the flag information indicates that the specificpicture in the reference layer is not used for the inter-layerprediction, decoding of the specific picture is ignored or skippedwithout affecting decoding of other pictures in the current layer. 22.The method of claim 20, wherein the value of the flag information isequal to one of 0 and
 1. 23. The method of claim 20, wherein when theflag information indicates that the specific picture in the referencelayer is not used for the inter-layer prediction, the specific picturein the reference layer is marked as unused for reference.
 24. The methodof claim 20, wherein the inter-layer reference picture is derived basedon pictures indicated by the flag information to be used for inter-layerprediction among pictures in the reference layer.
 25. The method ofclaim 24, wherein the specific picture in the reference layer indicatedby the flag information not used for the inter-layer prediction ismarked as unused for reference, and the inter-layer reference picture isderived from pictures which are not marked as unused for reference. 26.A video encoding method by a video encoder, comprising: encodingpictures in a reference layer; decoding and storing the pictures in thereference layer; deriving an inter-layer reference picture for a currentblock from at least one of the decoded pictures in the reference layer;constructing a reference picture list comprising the inter-layerreference picture in the reference layer and a reference picture in thecurrent layer; deriving a predicted sample of a current block in acurrent layer based on the reference picture list; deriving a residualsample of the current block based on the predicted sample and anoriginal sample of the current block; generating residual informationfor the residual sample of the current block; generating flaginformation for whether a picture in the reference layer is not used forthe inter-layer prediction; and generating a bitstream including theresidual information and the flag information, wherein when the flaginformation indicate that a specific picture in the reference layer isnot used for the inter-layer prediction, the specific picture is notcomprised in the reference picture set, and wherein the flag informationis transmitted through a slice segment header, and in all slice segmentheader of the picture in the reference layer, a value of the flaginformation is set to the same.
 27. The method of claim 26, wherein whenthe flag information indicates that the specific picture in thereference layer is not used for the inter-layer prediction, decoding ofthe specific picture can be ignored or skipped without affectingdecoding of other pictures in the current layer.
 28. The method of claim26, wherein the value of the flag information is equal to one of 0and
 1. 29. The method of claim 26, wherein when the specific picture inthe reference layer is not referred to in prediction of the currentblock and is not referred to in prediction of another picture in thereference layer which is referred to in prediction of the current block,the specific picture in the reference layer is marked as unused forreference, and the flag information comprises information specifying thespecific picture which is not referred to in prediction of the currentblock and is not referred to in prediction of another picture in thereference layer that is referred to in prediction of the current block.30. A non-transitory computer-readable storage medium storing abitstream comprising: residual information for a residual sample derivedbased on a predicted sample and an original sample of a current block;flag information for whether a picture in a reference layer is not usedfor an inter-layer prediction of the current block, wherein when theflag information indicate that a specific picture in the reference layeris not used for the inter-layer prediction, the specific picture is notcomprised in a reference picture set, and wherein the flag informationis received through a slice segment header, and in all slice segmentheader of the picture in the reference layer, a value of the flaginformation is set to the same.